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Webinar  Goals 


UL  4600:  Standard  for  Safety  for  the 
Evaluation  of  Autonomous  Products 

■  Overview  for  technical  stakeholders 

•  Comments  due  Friday  November  1 

■  Goals  for  this  Webinar 

•  Orientation  to  standard  for  technical  audience 

•  Key  principles  to  keep  in  mind  when  commenting 

•  How  to  get  a  copy  and  submit  comments 


Why  UL? 

■  Underwriters  Laboratories: 

working  for  a  Safer  World  for  1 25  years 


CELEBRATING 

125 


YEARS 


•  Published  first  safety  standard  in  1903 

•  Focus  on  research,  education,  and  more  than  1,700  standards 


■  UL's  Standards  Development  process 

•  Consensus  process 

•  Open,  transparent,  and  timely 

•  Continuous  standards  maintenance 


UL  4600  Standards  Technical  Panel  (STP) 

■  STP  is  the  voting  consensus  body 
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Timeline 


■  Initial  drafting 

•  July  201 8:  Announced  intent  to  develop  UL  4600 

■  STP  revisions 

•  June  201 9:  STP  meeting  to  discuss  first  full  draft 

•  Three  rounds  of  STP  comment  &  draft  revisions  completed 

■  Stakeholder  comments 

•  Oct  2019:  Stakeholder  preliminary  draft  available 

•  Stakeholder  comments  due  Nov  1, 2019 

■  Target  final  version  release  Q1  2020 
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Technical  Overview 

■  Orientation  to  current  preview  draft  version 

•  Contents  and  organization  subject  to  change! 

■  UL  4600  Scope 

•  Fully  Autonomous  Vehicle  (AV)  operation 

•  No  human  driver/supervisor 

■  Main  principles 

•  Safety  case  is  front  and  center 

■  Guide  to  review  &  comments 


EDGE  CASE 
RESEARCH 
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3  EDGE  CASE 
RESEARCH 


UL  4600  Key  Ideas 

■  Goal:  structured  way  to  argue  that  AV  sufficiently  safe 

•  Non-prescriptive,  safety  case  approach 

•  Trace  all  safety  goals  (claims)  to  evidence 

•  Checks  and  balances  (self-audit  and  independent) 

■  Monitoring  and  feedback 

•  Detect  invalid  assumptions  &  gaps  in  coverage 

■  System  Level  +  Life  Cycle  approach 

•  Includes  fault  recovery,  supply  chain  issues,  expected  misuse 

■  Reference  lists  to  improve  completeness 

•  Prompts  &  epistemic  defeaters  for  coverage  (#DidYouThinkofThat?) 

•  Ability  to  argue  that  some  prompts  aren’t  applicable  ©  2019  Philip  Koopman  7 


Why  UL  4600?  3  RESEARCH 

Autonomous  systems  have  unique  needs 

•  No  human  supervision,  non-determinism, ... 

•  This  version:  highly  automated  vehicles 

System  level  approach  needed 

•  Functional  safety,  SOTIF,  road  tests,  simulation  all  play  a  role 
-  But  need  a  framework  to  put  the  pieces  together 

•  Adapt  as  technology  evolves 

Cooperate  rather  than  compete 

•  Can  accept  work  products  from  ISO  26262,  ISO/PAS  21448,  etc. 

Goal:  guidance  on  “Is  system  engineering  rigor  sufficient?" 

©  2019  Philip  Koopman  8 


3  EDGE  CASE 
RESEARCH 


Goal  Based  Approach 

■  Traditional  safety  standards  are  prescriptive 

•  "Here  is  how  to  do  safety”  (process,  work  products) 

-  ISO  26262,  ISO/PAS  21448, 1  EC  61 508,  MIL-STD  882,  etc. 

•  But,  we’re  still  figuring  out  some  aspects  of  AV  safety 

■  UL  4600  is  goal  based:  “be  acceptably  safe" 

•  Use  a  Safety  Case  to  argue  system  is  acceptably  safe 

-  Define  what  safe  means;  argue  that  AV  meets  that  definition 

-  Do  NOT  prescribe  any  particular  engineering  approach 

-  DO  require  a  set  of  minimum  acceptable  topics  for  safety  case 

•  Require  use  of  any  good  system  engineering  process  (not  just  V) 
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What's  A  Safety  Case? 
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EDGE  CASE 
RESEARCH 


A  structured  argument  backed  by  evidence 

•  Notation  agnostic  /  use  any  reasonable  notation 

SubGoal/Claim:  "AV  will  not  hit  pedestrians" 

•  Hypothetical  Arguments 

-  "AV  will  detect  pedestrians  of  all  types” 

-  "AV  will  stop  or  avoid  collision  detected  pedestrians” 

-  "We  have  identified  &  mitigated  risks  caused  by 
difficult  to  detect  pedestrians" 

•  Hypothetical  Evidence 

-  "Here  are  results  of  detect  &  avoid  tests" 

-  "Here  is  analysis  of  coverage  of  different  types  of  pedestrians" 

-  "Reliability  growth  data  shows  high  pedestrian  coverage"  © 201 9 pinup Koopman  10 


UL  4600  Scope 
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EDGE  CASE 
RESEARCH 


System  level  safety  for  autonomous  operation  &  lifecycle 


SYSTEM  (Item  scope:  Vehicle  +  Infrastructure) 


ODD  SPECIFIED 


PROMPT  ELEMENTS  TAILORED  TO  ODD  &  SYSTEM 


RIGOROUS  DEVELOPMENT  PROCESSES 


RIGOROUS  OPERATIONAL  PROCESSES 


ADDRESSES  PROMPT  ELEMENTS 


TRACEABILITY  WITHIN  SAFETY  CASE  &  TO  UL4600 


REASONABLE  INDUCTIVE  STEPS  /  AVOIDS  PITFALLS 


METRICS  MONITOR  SAFETY  CASE  VALIDITY 


SELF-AUDITS 


INDEPENDENT  ASSESSMENT 


SAFETY  CULTURE 


CONTEXT 

DEFINED 


SAFETY  CASE 
WELL  FORMED 


/" 

HAZARDS 

IDENTIFIED 

TOP  LEVEL  GOAL: 

AV  SAFETY  CASE 

IS  ACCEPTABLE 
(Hypothetical/ 
Simplified) 

RISKS 

MITIGATED 


FAULT  MODELS  DEFINED 


VEHICLE  (SYSTEM  &  SOFTWARE) 


AUTONOMY  PIPELINE 


DATA,  NETWORKING,  SERVICES 


ROAD  USERS 


LIFE  CYCLE  &  SUPPLY  CHAIN 


MAINTENANCE  &  INSPECTIONS 


TOOLS  &  COMPONENTS 


HAZARDS  MAPPED  TO  RISK-BASED  INTEGRITY 


FAULT  RESPONSE  &  ODD  VIOLATION  STRATEGY 


MITIGATIONS  IDENTIFIED  &  SUFFICIENT 


L 


DEPENDABILITY  ISSUES  ADDRESSED 


FEEDBACK  TO  MANAGE  UNKNOWNS 
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Out  of  Scope  for  UL  4600  3  RESEARCH 


■  Related  topics 

•  ADAS  features 

•  AV  testing  safety  (but,  see  BSI/PAS  1881) 

•  Ethical  guidelines  (but,  see  IEEE  P7009) 

■  Human  factors 


Human  attention  (as  driver;  as  safety  supervisor) 

How  to  argue  humans  will  behave  as  required 
How  to  argue  human  safety  supervisor  will  react  correctly 


■  Details  of  security 

•  Requires  security  plan;  maps  security  plan  to  safety 

•  Does  not  attempt  to  define  what  is  in  security  plan 
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Prompt  Elements:  #DidYouThinkofThat?  0  research 


■  Extensive  lists  of  safety  case  topics,  hazards,  etc. 

•  Good  practices  &  Pitfalls  (lessons  learned  &  bad  practices  to  avoid) 

■  Prompts  must  be  considered,  not  necessarily  adopted 

•  Mandatory:  you  have  to  do  this 

•  Required:  can  deviate  ONLY  if  inherently  inapplicable 

-  E.g.,  if  no  machine  learning,  then  can  deviate  from  ML  requirements 

•  Highly  Recommended:  can  deviate  with  non-trivial  rationale 

•  Recommended:  entirely  optional 

•  Examples:  illustrative  reminders;  do  not  have  to  address  each  one 

■  Many  processes  and  technique  areas  are  lightly  constrained 

•  E.g.,  Identify  hazards,  but  use  any  reasonable  techniqu§201 


Operational  Design  Domain  (ODD)  3  RESEARCH 


■  Define  relevant  ODD  considering: 

•  Infrastructure 

•  Weather  &  road  conditions 

•  Object  &  event  ontology 

•  Own  and  other  vehicle  conditions 

•  ...  many  other  things 

■  Exiting  ODD  must  be  safe 

•  Due  to  environment  change  (unexpected  snow) 

•  Due  to  ODD  ontology  gap  ("what  the  heck  is  that???") 

•  Due  to  equipment  failure  (potentially  using  degraded  modes) 

©  2019  Philip  Koopman  14 


UL  4600  ODD  Prompt  Excerpts  3  RESEARCH 


Travel  infrastructure 

EXAMPLES:  types  of  road  surfaces,  road 
geometries,  bridge  restrictions 

Object  coverage  (i.e.,  objects  within  ODD) 

Event  coverage 

EXAMPLES:  interactions  with  infrastructure 

Behavioral  rules 

EXAMPLES:  traffic  laws,  system  path  conflict 
resolution  priority,  local  customs,  justifiable  rule 
breaking  for  safety 

Environmental  effects 

EXAMPLES:  weather,  illumination 

Vulnerable  populations 

EXAMPLES:  pedestrians,  motorcycles,  bikes, 
scooters,  other  at-risk  road  users,  other  road  users 

Seasonal  effects 

EXAMPLES:  foliage  changes,  sun  angle  changes, 
seasonally-linked  events  (e.g.,  Oktoberfest) 


■  Support  infrastructure,  if  any  is  relied  upon 

EXAMPLES:  types  of  traffic  signs,  travel  path 
geometry  restrictions,  other  markings 

■  Localization  support,  if  relied  upon 
EXAMPLES:  GNSS  availability,  types  of  navigation 
markers,  DSRC,  other  navaids 

■  Compliance  strategy  for  traffic  rules 
EXAMPLE:  enumeration  of  applicable  traffic 
regulations  and  ego  vehicle  behavioral  constraints 

■  Special  road  user  rules 

EXAMPLES:  bicycles,  motorcycles/lane  splitting, 
construction  systems,  oversize  systems, 
snowplows,  sand/salt  trucks,  emergency  response 
systems,  street  sweepers,  horse-drawn  systems 

■  Road  obstructions 

EXAMPLES:  pedestrian  zone  barriers,  crowd 
control  barriers,  police  vehicles  intentionally 
blocking  traffic,  post-collision  vehicles  and 
associate  debris,  other  road  debris,  other  artificial 
obstructions  ©  201 9  Philip  Koopman  1 5 


Autonomy 


EDGE  CASE 
RESEARCH 


■  Autonomy  Pipeline 

•  Sensing 

•  Perception 

•  Machine  learning 

•  Planning 

•  Prediction 

•  Trajectory  &  control 

•  Timing 


candidate  best  practices  &  pitfalls 

(e.g.,  correlated  sensor  faults) 

(e.g.,  brittle  perception,  ontology  gaps) 
(e.g.,  overfitting) 

(e.g.,  plan  exceeds  vehicle  capability) 
(e.g.,  mis-predictions,  sudden  changes) 
(e.g.,  degraded  vehicle  capabilities) 
(e.g.,  loss  of  control  loop  stability) 


System,  Environment,  Lifecycle  3  RESEARCH 


"Item"  covered  by  safety  case  includes  safety  related: 

•  Autonomy  (sensors,  algorithms,  actuators) 

•  Vehicle  (safety  related  within  autonomy  purview) 

•  Maintenance  and  inspection  procedures 

•  Lifecycle  issues  and  supply  chain 

•  Data  sources  and  feeds,  including  maps,  ML  training 


Assumptions  &  supporting  requirements 

•  ODD  characterization 

•  Road  infrastructure  support 

•  Procedural  support  (e.g.,  safety  related  inspections) 


*  ^4  7  OF  c* 
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Maintenance  &  Inspections  RESEARCH 

Safety  related  maintenance 

•  What  maintenance  is  required  for  safety? 

•  Are  procedures  documented? 

•  How  do  you  know  it  is  done  effectively? 

Safety  related  inspections 

•  What/when  are  inspections  required? 

•  Detection  of  vehicle  &  infrastructure  problems  (e.g.,  loose  wheel) 

•  Are  you  trusting  casual  passengers  with  life  critical  inspections? 

-  (Really?  Is  that  a  good  idea?) 
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Lifecycle  &  Supply  Chain  3  RESEARCH 


Item  has  valid  safety  case  at  all  times  once  deployed 
Safety  related  aspects  of  lifecycle 


•  Requirements/design/ML  training 

•  Handoff  to  manufacturing 

•  Manufacturing  &  deployment 

•  Supply  chain 

•  Field  modifications  &  updates 

•  Operation 

•  Retirement  &  disposal 

Update  distribution  &  integrity 


Is  sensor  cleaning  fluid  life  critical? 


•  Version  control  &  configuration  management 
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Role  of  Humans 

■  There  is  no  “captain  of  the  ship" 

•  Autonomy  must  assume  responsibility 

■  Interacting  with  people 

•  Occupants,  cargo  loading 

•  Pedestrians  &  mobility  device  users 

•  Other  drivers 

•  Special  populations 

•  Misuse,  pranks,  malfeasance 

■  Safety  related  lifecycle  participants 

•  Inspection  &  maintenance  accuracy 

■  Safety  culture  for  all  stakeholders 


EDGE  CASE 
RESEARCH 


Is  it  safe  to  drive  now? 
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Black  Swans  &  Unknowns  3  RESEARCH 

■  Inductive  proofs  are  never  complete 

•  The  black  swan  problem  - 

you  don't  know  what  you  don't  know 

■  Addressed  via: 

•  Extensive  use  of  prompts  for  better  coverage 

•  Epistemic  defeaters  (e.g.,  pitfalls) 

•  Monitoring  required  for  assumptions  and  unknowns 

■  Deploying  with  uncertainty 

•  You  will  deploy  believing  you  are  acceptably  safe 

•  Use  monitoring  to  reduce  margin  of  belief  uncertainty 

©  2019  Philip  Koopman  21 
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Every  observed  swan  is  white. 
Therefore  all  swans  are  white. 


Assessment:  T rust  and  Verify 


EDGE  CASE 
RESEARCH 


■  Self-audit 

•  Audit  safety  case  for  completeness 

•  Check  technical  aspects  for  reasonableness 

•  In  close  collaboration  with  the  development  team 


■  Independent  assessor 

•  Independence  from  developer  &  competence  must  be  documented 

•  Check  and  balance  on  self-audit 

•  NOT  expected  to  find  technical  defects 

■  Developers  must  "own"  safety 

•  Audits  &  assessments  serve  as  a  check  and  balance 
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Feedback  Loops 


EDGE  CASE 
RESEARCH 


Feedback  used  to  mitigate  risk  of  unknowns 


•  Within  product:  incidents  trigger  safety  case  update 

•  At  Assessment:  updates  trigger  assessments 


Standards  Process:  emergent  issues  trigger  ~yearly  standard  update 


*ppi i. 


SAFETY  CASE 


EXPERIENCE/^™0™™™0 

-  Validation  ~~£§e, 

-  Testing 

-Deployment  I  V*.E'|DENC^ 

/  FAULTS  ASSfc^2 


UL  4600 
STANDARD 


FAULTS  IN 
STANDARD 


GAPS  IN 
STANDARD 


STAKEHOLDERS 
&STP 
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Component  Assessment  3  RESEARCH 


Idea:  design-by-contract 
component  interface 

•  Assured  properties  (services;  functions) 

•  Assumptions  made  by  component 

-  Must  match  promises  made  by  system 

•  Component  assurance  context 

-  Fault  model 

-  Subset  of  UL  4600  clauses  assessed 

•  Can  assess  SEooC  conformance  independent  of  system 
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Generalized  idea  of  System  Element  out  of  Context  (SEooC) 

•  Hardware  and/or  software 


Change  &  Impact  Analysis  3  RESEARCH 

■  Continual  changes 

•  System  functionality  update 

•  Different  ODD  (changing  ODD  scope;  surprises) 

■  Assessment  in  response  to  changes: 

•  Impact  analysis 

•  If  required:  Update  safety  case 

•  If  safety  case  updated:  Update  self-audit 

•  If  "big"  safety  case  change:  Independent  Assessment  update 

■  “Size"  of  change  relates  to  safety  case,  not  lines  of  code 

•  Impact  analysis  informs  scope  of  self-audit/assessments 
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Prompt  Elements  vs.  Integrity  Levels  3  RESEARCH 


Prompt  element  deviation  categories: 

•  Mandatory  /  Required  /  Highly  Recommended  /  Recommended 

-  E.g.:  "REQUIRED"  can  only  deviate  if  intrinsically  inapplicable 

Integrity  levels 

•  Define  at  least  two  integrity  levels:  life  critical  &  injury 

-  OK  to  adopt  more  and/or  existing  levels  (e.g.,  ASIL,  SIL,  DAL) 

•  Define  level  of  rigor/technique  use  based  on  integrity  level 

Example:  Static  analysis 

•  Required  that  static  analysis  is  used  to  some  degree 

•  Coverage,  tools,  tool  settings  based  on  Integrity  level 
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How  UL  4600  Works  with  Others  3  RESEARCH 

■  ISO  26262  -  starting  point 

•  Still  relevant  to  the  extent  it  can  be  applied 

•  Assumes  traceability  of  tests  to  design  with  "V" 

■  ISO/PAS  21448  &  SaFAD  -  more  guidance 

•  Design  and  validation  process  framework 

■  UL  4600  -  #DidYouThinkofThat? 

•  Provides  a  template  for  technical  safety  report 

•  Minimum  criteria  for  complete  coverage  +  feedback  requirement 

•  Lists  of  positive  and  negative  lessons  learned 

•  Objective  assessment  criteria  for  safety  case 
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UL  4600  Chapter  Short  Titles  3  RESEARCH 


■  Organized  by  practitioner  skill  set 


1 .  Preface 

2.  Scope 

3.  References 

4.  Terms 

5.  Safety  case  &  arguments 

6.  Risk  assessment 

7.  Humans  &  road  users 

8.  Autonomy 


9.  Software  &  system 
engineering 

10.  Dependability 

1 1  Data  &  networking 

12.  Verification  &  validation 

13.  Tool  qualification 

14.  Lifecycle  concerns 

15.  Maintenance 

1 6.  Metrics 

17.  Assessment 
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Anticipated  UL  4600  Technical  Benefits  3  RESEARCH 


■  Catalog  of  best  practices:  #DidYouThinkofThat? 


Avoid  missed  hazards 
Avoid  pitfalls 

Mechanism  for  industry  to  share  without  sharing  detailed  data 


■  Objective,  repeatable  independent  assessment 

•  Self-audit  is  first  level  of  checks  and  balances 

-  Feedback  identifies  surprises/gaps 

•  Independent  assessment  is  about  well-formed  safety  case 

-  Not  subjective  opinion  about  whether  developer  tried  hard  enough 

-  Prompt  elements  provide  a  safety  case  coverage  floor 

-  But,  developer  assumes  burden  for  safety 
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Get  Involved:  Submit  Comments 

■  Commenting  requires  registering  as  stakeholder 

•  E-mail  to:  <Deborah.Prince(g)ul.com> 

■  Use  supplied  spreadsheet  for  consideration 

•  Please  make  as  concrete  &  actionable  as  possible 


Reviewing  Organization:  PUT  YOUR  ORGANIZATION  HERE 

Point  of  Contact:  PUT  YOUR  NAME  and  e-mail  address  HERE;  please  combine  comments 


# 

Page 

Clause 

Old  text 

New  text 

Discussion 

1 

54 

5.2.3.3.C.1 

Quote  the  old  text 
before  change 

Your  proposed  new 
text  with  change 

Explain  (could  be  just 
"typo"  or  "format"  if 
that  is  the  issue). 

2 

3 
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Comments  &  Timeline 


■  Official  version  &  comment  spreadsheet  via  UL  CSDS 

•  Other  public  materials  and  draft  at:  UL4600.com 

■  Timeline: 

•  Comments  due  Friday  Nov  1st  via  CSDS  upload 

•  Potentially  voting  draft  in  December 

•  Target  for  approved  standard:  Q1  2020. 

■  Will  Stakeholder  names  be  public? 

•  Stakeholder  list  itself  is  private 

•  However,  all  preliminary  review  comments  are  public  &  attributed 

to  commenter 


